Development and Validation of a Scoring Rubric for Editorial Evaluation of Peer-review Quality: A Pilot Study.

Introduction: Despite the importance of peer review to publications, there is no generally accepted approach for editorial evaluation of a peer review's value to a journal editor's decision-making. The graduate medical education editors of the Western Journal of Emergency Medicine Special Issue in Educational Research & Practice (Special Issue) developed and studied the holistic editor's scoring rubric (HESR) with the objective of assessing the quality of a review and an emphasis on the degree to which it informs a holistic appreciation for the submission under consideration. Methods: Using peer-review guidelines from several journals, the Special Issue's editors formulated the rubric as descriptions of peer reviews of varying degree of quality from the ideal to the unacceptable. Once a review was assessed by each editor using the rubric, the score was submitted to a third party for blinding purposes. We compared the performance of the new rubric to a previously used semantic differential scale instrument. Kane's validity framework guided the evaluation of the new scoring rubric around three basic assumptions: improved distribution of scores; relative consistency rather than absolute inter-rater reliability across editors; and statistical evidence that editors valued peer reviews that contributed most to their decision-making. Results: Ninety peer reviews were the subject of this study, all were assessed by two editors. Compared to the highly skewed distribution of the prior rating scale, the distribution of the new scoring rubric was bell shaped and demonstrated full use of the rubric scale. Absolute agreement between editors was low to moderate, while relative consistency between editor's rubric ratings was high. Finally, we showed that recommendations of higher rated peer reviews were more likely to concur with the editor's formal decision. Conclusion: Early evidence regarding the HESR supports the use of this instrument in determining the quality of peer reviews as well as its relative importance in informing editorial decision-making.

Development of entrustable professional activities for emergency medicine medical education fellowships: A modified Delphi study.

Background: It is essential that medical education (MedEd) fellows achieve desired outcomes prior to graduation. Despite the increase in postgraduate MedEd fellowships in emergency medicine (EM), there is no consistently applied competency framework. We sought to develop entrustable professional activities (EPAs) for EM MedEd fellows. Methods: From 2021 to 2022, we used a modified Delphi method to achieve consensus for EPAs. EM education experts generated an initial list of 173 EPAs after literature review. In each Delphi round, panelists were asked to make a binary choice of whether to include the EPA. We determined an inclusion threshold of 70% agreement a priori. After the first round, given the large number of EPAs meeting inclusion threshold, panelists were instructed to vote whether each EPA should be included in the "20 most important" EPAs for a MedEd fellowship. Modifications were made between rounds based on expert feedback. We calculated descriptive statistics. Results: Seventeen experts completed four Delphi rounds each with 100% response. After Round 1, 87 EPAs were eliminated and two were combined. Following Round 2, 46 EPAs were eliminated, seven were combined, and three were included in the final list. After the third round, one EPA was eliminated and 13 were included. After the fourth round, 11 EPAs were eliminated. The final list consisted of 16 EPAs in domains of career development, education theory and methods, research and scholarship, and educational program administration. Conclusions: We developed a list of 16 EPAs for EM MedEd fellowships, the first step in implementing competency-based MedEd.

An Achilles Tendinopathy Educational Intervention for Generalist Physicians Raises Awareness and Improves Knowledge for Treating Ballet Dancers.

BACKGROUND: Achilles tendinopathy can be a devastating condition in ballet dancers. Initial medical evaluation is frequently conducted by generalist physicians, who may lack the perspective of dance and sports medicine focused training and often prescribe complete rest for chronic overuse injuries. In order to provide targeted information about Achilles tendinopathy and the unique needs of elite athletes, such as ballet dancers, we designed an educational intervention that covers relevant anatomy, physiology, biomechanics, clinical presentation and evaluation, and recommendations for initial treatment and sub-specialty referral. The objectives of this study were to determine baseline understanding of Achilles tendinopathy by generalist physicians and to evaluate the efficacy of an educational intervention on their knowledge. METHODS: This was a prospective pre- post-test study, to measure improvement in knowledge about Achilles tendinopathy. Emergency Medicine and Primary Care physicians participated in an educational intervention with didactic and hands-on components. Data were analyzed using a two-tailed t-test. RESULTS: Twenty generalist physicians participated. There was significant improvement in their knowledge of Achilles tendinopathy for all our educational objectives (Pre-test: 59% correct answers; post-test: 79% correct answers; P < .0001; 95% CI: 6.32-10.78.). CONCLUSIONS: A brief educational intervention leads to increased knowledge of Achilles tendinopathy in generalist physicians. By increasing awareness of Achilles tendinopathy through a standard educational model, we propose that our curriculum can be portable and accessible to all health care providers who encounter dancers and other athletes with Achilles tendinopathy and may serve as a model for expanded education to generalists about other conditions commonly encountered in ballet dancers.

A guide to creating an educator's portfolio for the 21st century.

An educator's portfolio is (EP) a collection of materials for demonstrating and assessing a clinician educator's performance and perspective. Although not all academic institutions require faculty to maintain an EP, it can serve as a valuable tool for both personal reflection and professional advancement. With newer advancements in technology and social media, there are also opportunities to enhance the EP for the digital era. This educator's blueprint highlights eight strategies for creating an EP for the 21st century clinician educator: use your educator's philosophy to introduce and anchor your EP, apply broad definitions of scholarship, include pertinent metrics for all scholarly products, describe mentorship activities, incorporate self-assessment and the assessments from others, report comprehensive accounting of presentations, highlight leadership activities, and utilize technology to facilitate dissemination and sharing of your EP.

A Collaborative Approach to Mentored Peer Reviews Sponsored by the Council of Residency Directors in Emergency Medicine.

Introduction: Historically, there have been no systematic programs for teaching peer review, leaving trainees to learn by trial and error. Recently, a number of publications have advocated for programs where experienced reviewers mentor trainees to more efficiently acquire this knowledge. Objective: Our goal was to develop an introductory learning experience that intentionally fosters peer-review skills. Methods: The Council of Residency Directors in Emergency Medicine (CORD) offered education fellowship directors the opportunity to mentor their fellows by reviewing submitted manuscript(s) supplemented by educational material provided by their journal. Reviews were collaboratively created. The decision letter that was sent to manuscript authors was also sent to the mentees; it included all reviewers' and editor's comments, as feedback. In 2022, fellows received a post-experience survey regarding prior experiences and their perspectives of the mentored peer-review experience. Results: From 2020-2022, participation grew from 14 to 30 education fellowships, providing 76 manuscript peer reviews. The 2022 survey-response rate of 87% (20/23) revealed that fellows were inexperienced in education scholarship prior to participation: 30% had authored an education paper, and 10% had performed peer review of an education manuscript. Overall, participants were enthusiastic about the program and anxious to participate the following year. In addition, participants identified a number of benefits of the mentored experience including improved understanding of the scholarship process; informing fellows' scholarly pursuits; improved conceptualization of concepts learned elsewhere in training; and learning through exposure to scholarship. Conclusion: This program's early findings suggest that collaboration between academic societies and interested graduate medical education faculty has the potential to formalize the process of learning peer review, benefitting all involved stakeholders.

From inequity to access: Evidence-based institutional practices to enhance care for individuals with disabilities.

People with disabilities experience barriers to care in all facets of health care, from engaging with the provider in a clinical setting (attitudinal and communication barriers) to navigating a large institution in a complex health care environment (organizational and environmental barriers), culminating in significant health care disparities. Institutional policy, culture, and physical layout may be inadvertently fostering ableism, which can perpetuate health care inaccessibility and health disparities in the disability community. Here, we present evidence-based interventions at the provider and institutional levels to accommodate patients with hearing, vision, and intellectual disabilities. Institutional barriers can be met with strategies of universal design (i.e., accessible exam rooms and emergency alerts), maximizing electronic medical record accessibility/visibility, and institutional policy development to recognize and reduce discrimination. Barriers at the provider level can be met with dedicated training on care of patients with disabilities and implicit bias training specific to the surrounding patient demographics. Such efforts are crucial to ensuring equitable access to quality care for these patients.

ChatGPT and conversational artificial intelligence: Friend, foe, or future of research?

Artificial intelligence (AI) and machine learning are increasingly utilized across healthcare. More recently, there has been a rise in the use AI within research, particularly through novel conversational AI platforms, such as ChatGPT. In this Controversies paper, we discuss the advantages, limitations, and future directions for ChatGPT and other forms of conversational AI in research and scholarly dissemination.

The next generation of researchers: One-year outcome data from the SAEM Advanced Research Methodology Evaluation and Design in Medical Education (ARMED MedEd) program.

Introduction: As the field of medical education evolves, there is a need to increase the quality of education scholarship and develop a cadre of research scholars; however, clinician educators (CEs) considering this career transition have limited formal training in education research methodology to heed this call. Therefore, a program that provides more advanced training in education scholarship for CEs without the financial and resource barriers of fellowships and masters programs is needed. Methods: The SAEM Advanced Research Methodology Evaluation and Design in Medical Education (ARMED MedEd) program is a longitudinal program for the beyond-beginner CE, seeking advanced training in education research. The program was created using a comprehensive needs assessment and included longitudinal training; small-group projects; dedicated project mentors; and integrated diversity, equity, and inclusion initiatives. Program participants applied for a grant upon program completion. Results: Twenty-one participants completed the course with 100% completing the baseline survey and 67% (14/21) completing the end-of-program survey. Participants reported improved perception of knowledge across all of the topics with a medium to large effect size, ranging from 0.40 to 0.62. When asked about impact on their network of potential collaborators, participants reported a median of 7 (interquartile range [IQR] 5-8) out of 9. When asked about the impact on their community of practice, participants reported a median of 7 (IQR 5-7) out of 9. When asked about the impact on their professional identity, participants reported a median of 7 (IQR 4-9) out of 9. Participants also reported an increase in both the quantity (mean of 2 ± 1 new mentors) and the quality (median score 7 [IQR 5-8] out of 9) of new research mentorship as a result of the program. Open-ended feedback was generally positive, with 100% reporting they would advise others to take this program. Conclusions: The SAEM ARMED MedEd program represents a proof of concept for an advanced education research program seeking to fill the research training gap for the beyond-beginner Clinician educators.

Promoting Access and Equity: A Historical Perspective of Healthcare Access for People With Disabilities.

People with disabilities represent a large and often under-recognized minority population in the United States. Historically, negative healthcare provider perceptions and limited critical social determinants of health (including community living and education) have resulted in inequitable healthcare and access for this vulnerable group. Within the last 40 years, there have been some advances in legislation to improve access and support for those with disabilities. Since then, advances in accommodations have enabled better access to critical health-related resources and care. Continued forward progress and increased awareness are imperative to improve access, reduce disparities in healthcare, and combat discrimination.

Faculty Development for Academic Emergency Physicians: A Focus Group Analysis.

OBJECTIVES: The objective is to explore academic emergency medicine physicians' exposure to and needs regarding faculty development. METHODS: We conducted a prospective qualitative study of Society for Academic Emergency Medicine members in 2018 using focus groups selected by convenience and snowball sampling. One facilitator ensured representative engagement and responses were transcribed in real-time by an assistant after obtaining verbal consent. Results were analyzed using a grounded theory approach with a constructivist perspective. Thematic analysis was refined using the constant comparative method. RESULTS: Sixteen physicians participated in the focus groups, representing a diverse group of perspectives. Six themes emerged about unmet needs in faculty development: knowledge and skills, relationships, specific programs or resources, and professional benefits. CONCLUSIONS: Members of a national academic society identified three areas of focus important to developing academicians in emergency medicine: content for faculty developers, relationship-building among members, and support from the organization as a "professional home." Academic societies can use this to guide future programming.

Making emergency medicine accessible for all: The what, why, and how of providing accommodations for learners and physicians with disabilities.

Individuals with disabilities comprise a substantial portion of the U.S. population but make up only a small subset of medical students and health care providers. Both the Association of American Medical Colleges and the Accreditation Council for Graduate Medical Education have called for increased diversity in the physician workforce, to more closely represent the U.S. patient population and provide culturally effective care. Yet the barriers to disclosure and inclusion for individuals with disabilities in health care are significant, including attitudinal barriers such as stigma and bias, organizational barriers in policies and procedures, and environmental barriers such as resources and physical space. Lack of experience providing accommodations and a lack of knowledge of both what is legally required and what is possible also prevent programs from creating access. Realizing inclusion for individuals with disabilities in a diverse workforce requires emergency medicine programs to be proactive and deliberate in their approach to recruiting, accommodating, and retaining students, residents, and faculty with disabilities. Such efforts are likely to provide benefits that extend beyond those who receive the accommodations.

Promotion and tenure letters: A guide for faculty.

Promotion and tenure (P&T) letters are a key component of the academic advancement portfolio. Despite their importance, many faculty are not trained to write these letters and there is limited literature describing the approach and key components. This paper reviews the role of P&T letters and provides general guidelines for writers. We present a step-by-step guide, which includes how to respond to requests, the role of institutional guidelines, providing context to the letter, evaluating candidates, and delivering an overall recommendation. Finally, we discuss current controversies in P&T letters. This paper is intended to help novice and more experienced writers to enhance their P&T letters, while also helping applicants for promotion understand what is being asked of their letter writers.

Beyond the CLAIM: A comprehensive needs assessment strategy for creating an Advanced Medical Education Research Training Program (ARMED-MedEd).

BACKGROUND: The health professions education (HPE) landscape has shifted substantively with increasing professionalization of research and scholarship. Clinician educators seeking to become competitive in this domain often pursue fellowships and master's degrees in HPE, but there are few resources for the continuing professional development (CPD) of those who seek to continue developing their scholarly practice within HPE. Acknowledging the multiple players in this landscape, the authors sought to design a new "beyond beginners" HPE research program using a novel needs assessment planning process. METHODS: The authors developed and conducted a new three-phase, five-step process that sets forth a programmatic approach to conducting a needs assessment for a CPD course in HPE research. The five steps of the CLAIM method are: Competitive analysis, Literature review with thematic analysis, Ask stakeholders, Internal review by experts, and Mapping of a curriculum. These steps are organized into three phases (Discovery, Convergence, and Synthesis). RESULTS: Over a 12-month period, the authors completed a comprehensive needs assessment. The CLAIM process revealed that longitudinal digital connection, diverse and in depth exposure to HPE research methods, skills around scholarly publishing, and leadership and management of research would be beneficial to our design. CONCLUSIONS: The CLAIM method provided scaffolding to help the authors create a robust curriculum that adopts a scholarly approach for developing a HPE research course. This needs assessment methodology may be useful in other CPD contexts.

Planning for diverse, equitable, inclusive research in health professions education: An integral thread in the ARMED MedEd research course.

Racism in medicine affects patients, trainees, and practitioners and contributes to health care inequities. An effective strategy to actively oppose the structural racism ingrained in the fabric of medicine is to intentionally and systematically address diversity, equity, and inclusion (DEI) in medical education and research. As part of ARMED MedEd, a new longitudinal cohort course in advanced research methods in medical education, sponsored by the Society for Academic Emergency Medicine, the leadership team deliberately included a nested DEI curriculum. The goal of the DEI curriculum is to reduce bias in development, recruitment, and implementation of education research studies to promote equity and inclusion in medical education, research, and ultimately, patient care. A team of medical educators with expertise in DEI developed curricular elements focusing on DEI in education research. The two major components are a didactic curriculum (including implicit bias training) to teach researchers to consider equity as they design studies and a consultative service to refine research protocols to address lingering unintended bias. A dedicated focus on DEI can be incorporated into an advanced education research methodology course to raise awareness and provide tools to avoid bias in research design and implementation of interventions. Over time, the network of education researchers who are trained in DEI awareness will grow and provide equitable offerings to their learners to mitigate health inequities.

Moving beyond personal factors: A national study of wellness interventions in emergency medicine residency programs.

BACKGROUND: In 2017, the Accreditation Council for Graduate Medical Education (ACGME) revised its Common Program Requirements to support trainees and faculty by mandating programs to provide dedicated wellness resources and education. Emergency medicine may benefit from this change due to high burnout rates within the specialty. However, the current state of wellness interventions in emergency medicine (EM) residency programs has not yet been well described. Understanding current practices is necessary to assess unmet needs and inform the development and evaluation of future interventions that aim to improve trainee wellness. OBJECTIVES: The goal of this study was to describe currently implemented wellness interventions in EM residency programs. METHODS: This descriptive study surveyed 250 ACGME-accredited EM residency programs between March 1 and June 1, 2020, regarding wellness interventions. The survey included demographic questions; structured multiple-choice questions about cost, frequency, and champions; and free-text response options to briefly describe interventions. Respondents were also asked to classify the interventions according to the seven factors described in the National Academy of Medicine Model of Clinician Well-Being and Resilience. RESULTS: Ninety of 250 (36% response rate) residency programs participated, describing 162 unique wellness interventions. Respondents classified the majority of interventions (n = 136) as targeting personal factors according to the National Academy of Medicine model. Qualitative analysis revealed five major themes describing the interventions: program culture, program factors, environmental and clinical factors, wellness activities and practices, and wellness resources. CONCLUSIONS: Results of this survey may help to inform a national needs assessment addressing the current state of wellness interventions in EM residency programs. Our results highlight the need for more interventions targeting external factors impacting resident wellness.

A practical guide for conducting qualitative research in medical education: Part 3-Using software for qualitative analysis.

The process of performing qualitative analysis can be a daunting task. Technology can be employed to ease the burden of the work; however, the researcher may not fully appreciate how and when computer software can assist in conducting qualitative analysis. In this, the third installment of our "how-to" series on qualitative research methods, we describe basic concepts and approaches to using both simple word processing programs and specific qualitative research software programs to assist in coding and analysis. We hope that the concepts put forth in this paper will help qualitative researchers become more familiar with available technological approaches and that they will, in turn, enhance the efficiency of the research process as well as the depth, clarity and richness of research findings.